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A  Recommendation  Blank 

To  Aid  in  the  Selection  of 
Scientific  and  Technical  Personnel 

A.  INTRODUCTION 

1.  Selection  and  placement  officers  frequently  have  commented 
on  the  lack  of  value  of  the  usual  letters  of  recommendation  and  refer¬ 
ence  forms.  Persons  referred  to  by  applicants  are  apt  to  adhere  to  a 
safe  middle  ground  of  innocuous  and  generally  favorable  comments  in 
describing  the  applicant  or  to  give  extremely  high  evaluation,  even 
when  the  limitations  of  the  recommendee  are  known.  Letters  of  reeoa- 
mendation  are  too  frequently  characterized  by  generalities  and  stereo¬ 
typed  phrases  with  little  or  no  information  useful  in  evaluating  the 
applicant's  fitness  for  the  job.  In  a  few  cases,  of  course,  the  recom- 
mender  is  so  well  known  to  the  person  responsible  for  selection  that 

the  statements  made  about  the  applicant  are  highly  useful.  Nevertheless, 
the  ordinary  references  cannot  be  uniformly  interpreted. 

2.  To  meet  the  need  for  usable  references  on  applicants  for 

scientific  and  technical  positions ,  ^the  American  Council  on  Education, 
under  contract  with  the  Office  of  Naval  Research,  made  arrangements 
with  Richardson,  Bellows,  Henry  and  Company  to  develop  and  validate  an 
objective  and  scorable  reference  form.  Three  forms  of  objectively 
scored  recommendation  blanks  were  constructed  and  validated.  {See 
Appendix  A.)  t 

3.  Two  conparable  forms,  1-F  and  2-F,  use  the  forced-choice 
method.  The  other  form,  3-N  (non-forced-choice),  was  organized  in  a 
check  list  form.  All  three  forms  discriminated  well  between  scientists 
rated  by  their  superiors  as  "very  effective"  and  those  rated  as  "mini¬ 
mally  effective."  Sid  table  scoring  keys  were  constructed  for  each  form. 
Most  persons  will  be  able  to  complete  the  objective  recommendation  form 
for  an  applicant  in  less  than  twenty  minutes. 

b.  wsmmam 

1.  The  use  of  the  procedures  developed  in  this  study  should  be 
restricted  to  obtainf  ig  estimates  of  competence  of  applicants  for  phys¬ 
ical  science  positions  in  research  organizations.  Their  use  for  other 
classes  of  applicants  may  be  misleading.  While  the  material  may  be  of 
value  in  other  situations,  further  studies  should  be  made  in  order  to 
know  the  discriminative  value  and  to  set  standards  of  minimum  accept¬ 
ability. 
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2.  These  rating  forms  should  not  be  used  as  the  sole  means  of 
obtaining  estimates  of  an  applicant's  performance  under  former  super¬ 
visors  and  professors.  The7  should  be  only  a  part  of  a  more  compre¬ 
hensive  recommendation  blank  which  would  include  other  questions  or 
ratings,  as  well  as  the  "common  denominator"  of  the  forced-choice  or 
check-list  sections.  The  additional  questions  or  ratings  will  provide 
additional  information  as  to  the  particular  strengths  and  weaknesses 
of  applicants.  There  should  be  questions  pertaining  to  his  technical 
and  background  experience  in  the  area,  his  ability  to  work  within  a 
set  of  policies,  potential  levtl  of  development,  his  ability  to  get 
along  with  professional  and  social  associates,  his  ingenuity  and  crea¬ 
tiveness,  etc.  Such  data  are  needed  for  adequate  placement  of  the 
individual  employee.  (See  Appendix  A  for  suggested  forms.) 


3.  Possession  or  knowledge  of  the  scaring  keys  must  be  re¬ 
stricted  to  those  offices  in  which  credentials  are  evaluated.  As  soon 
as  recammendors  gain  knowledge  of  the  scoring  key,  the  value  of  these 
forms  wil  decrease  rapidly.  On  request  to  the  Human  Resources  Division, 
Office  of  Naval  Research,  copies  of  the  scoring  keys  will  be  made  avail¬ 
able  to  those  who  are  planning  to  use  these  forms. 


C. 
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1.  Recommendations  are  secured  in  order  to  determine  the  ability 
of  an  applicant  on  former  jobs.  The  problem  is,  therefore,  quite  similar 
to  that  of  evaluating  the  performance  of  employees.  The  research  in 
this  study  was  based  upon  extensive  studies  of  work  performance  evalua¬ 
tion.  These  studies  were  based  on  certain  assumptions  and  principles 
that  have  been  carefully  tested  in  practice.  The  basic  ideas  are? 


a.  Host  people  can  describe  a  subordinate  quite  accurately,  but 
cannot  be  trusted  to  evaluate  their  performance  objectively. 

It  is,  therefore,  desirable  to  separate  reporting  on  job 
performance  from  the  evaluation  of  that  performance. 

b.  language  used  in  a  performance  report  is  most  effective  when 
it  conforms  to  that  of  the  persons  reporting.  The  language 
used,  therefore,  is  drawn  directly  from  typical  letters  and 
reports. 

c.  The  significance  of  all  reported  aspects  of  job  performance 
most  be  carefully  checked  by  statistical  methods  in  order  to 
evaluate  them  properly. 

d.  Opportunity  must  be  given  to  any  recommend or  to  say  favor¬ 
able  things  about  the  person  on  idiom  he  is  reporting,  with 
a  chance  to  describe  differences  in  people.  Otherwise,  no 
usable  evaluations  of  individuals  are  possible. 

2.  The  reference  or  recommendation  forms  presented  in  this  study 
were  constructed  and  validated  on  the  same  basis  as  are  the  more  objec¬ 
tive  performance  reports.  In  faot,  a  recommendation  is  a  performance 
report  whioh  a  selection  officer  may  use  as  one  evidence  of  the  qualifi¬ 
cations  of  an  applicant  for  a  job.  A  brief,  selected  bibliography  on 
recommendation  blanks  end  on  forced-choice  rating  is  given  in  Appendix  D. 
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3.  There  are  distinct  phases  or  steps  in  the  development  of 
eoorable  recommendation  blanks.  After  defining  the  slasa  of  persons 
to  shoe  the  prooedure  is  to  be  applied ,  the  steps,  briefly,  are  as 
follows* 

a.  Amassing  the  words  and  phrases  which  are  used  in  describing 
individuals  with  reference  to  their  performance  or  probable 
performance  in  the  kinds  of  jobs  included .  Four  sources  of 
such  descriptive  material  were  available  for  this  job. 

1)  Descriptions  written  by  supervisors  of  more  effective 
and  less  effective  physical  scientists  employed  by  a 
large  industrial  research  department. 

2)  The  study  "Critical  Requirements  for  Research  Personnel* 
recently  completed  for  the  Office  of  Naval  Research  and 
the  American  Council  on  Education  (based  on  observations 
and  interviews  in  Naval  research  organizations) . 

3)  Letters  of  recommendation  written  by  scientists  in 
regard  to  scientists. 

4)  The  recommendation  blanks  for  contestants  in  the  Eighth 
Annual  Science  Talent  Search  conducted  by  Science  Service. 

b.  The  approximately  600  descriptive  words  and  phrases  obtained 
were  organized  into  three  forms  (about  200  wards  and  phrases 
each)  of  the  Inf  carnation  Sheets.  (See  Appendix  B  for  sample 
of  an  Information  Sheet  and  covering  letter.)  These  were 
sent  to  the  heads  of  departments  of  physics,  chemistry,  and 
engineering  in  universities — since  they  frequently  recommend 
persons  for  scientific  positions.  Each  department  chairman 
was  asked  to  accomplish  the  throe  forms— on  the  farm  stamped 
■best,*  rate  one  of  his  best  advanced  students;  on  the 
foam  stamped  *poor,*  rate  one  of  his  minimally  acceptable 
students;  and  on  the  unstamped  form  to  indicate  how  favorable 
each  item  was  as  a  description  of  a  scientist.  Two  hundred 
fifty  blanks  were  returned  and  used  in  this  study.  These 
provided  the  basis  far  the  selection  of  the  items  to  be 
used,  for  their  patterns  of  organization  into  blocks,  and 
for  the  construction  of  the  scoring  keys.  On  this  basis, 
the  semi-final  copy  for  Forms  E  and  F  ( forced-choice*)  and 
Foam  G  (nan-forced-choice)  were  constructed. 

e.  Supervisors  of  scientists  in  several  Navy  research  agencies, 
the  Bureau  of  Standards,  and  the  Brookhaven  National  Institute 

Sovided  the  validation  sample.  The  three  semi-final  forms 
,  F,  and  G)  were  randomly  arranged  in  padre  (E  and  F,  E  and 
0,  and  F  and  G).  Each  supervisor  was  asked  to  oomplete  one 


■Foroed-ohoioe  refers  to  the  fact  that  the  person  filling  in  the  form 
Is  required  to  amke  a  response  relative  to  a  limited  group  of  Ghoioes. 
This  usually  is  of  the  fan  *of  these  five  descriptive  statements,  mark 
with  am  Z  that  one  wbroh  is  most  descriptive  of  the  person  being  rated, 
lark  with  an  t  tha4  s  i tenant  which  is  least  descriptive  of  the  person 
being  rated." 

«*  • 
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pair  of  these  fonts,  one  for  one  of  his  best  scientists  and 
one  far  one  of  his  minimally  acceptable  scientists  in  accord¬ 
ance  with  a  letter  of  directions  accompanying  the  ferns.* 

(See  Appendix  C.)  On  the  basis  of  these  responses* 

1)  the  preliminary  keys  were  cross-validated,  i.e.,  checked 
by  testing  on  a  second  group, 

2)  the  least  effective  materials  in  Farms  E  and  F  were 
eliminated,  leaving  20  groups  or  blocks  of  items  in 
each  form, 

3)  revised  scoring  keys  were  developed  for  Forms  E  and  F 
(in  final  forms,  1-F  and  2-F) .  These  still  need  to  be 
tested  on  another  sample. 

Ho  information  was  available  as  to  the  size  of  the  group  from 
which  these  men  were  selected,  nor  was  it  possible  to  deter¬ 
mine  how  representative  were  these  groups  when  compared  with 
the  population  of  physical  scientists  as  a  whole.  However, 
the  results  of  the  study  would  indicate  that  the  scientists 
reported  upon  in  this  research  are  sufficiently  representative 
of  the  physical  scientists  employed  by  the  Navy  in  research 
laboratories  that  the  recommendation  procedures  can  be  con¬ 
sidered  as  quite  useful  to  aid  in  selecting  such  scientists 
for  the  Navy.  Further  validity  studies  should  be  made  in 
order  to  determine  the  applicability  of  the  procedures  to 
other  areas. 


D. 


1.  Budgetary  procedures  required  that  this  study  be  completed 
within  rigid  time  limits,  necessitating  certain  deviations  from  the 
standard  procedures  used  in  constructing  and  validating  such  farm.  These 
modifications  did  not  seriously  affect  the  validity  of  the  study,  but  they 
do  make  it  impossible  to  report  validity  indices  on  a  basis  comparable  to 
other  studies. 

2.  To  show  the  discrimination  effected  by  these  forms,  scares 
obtained  for  Forms  1-F  and  2-F  are  shown  in  Table  1.  The  difference  in 
sooroe  for  the  "best"  and  "poor*  scientists  shows  little  overlapping. 

If  a  random  sample  of  scientists  had  been  used,  the  range  of  scores 
would  have  been  about  the  same,  but  with  many  more  scores  around  the 
middle  of  the  distribution.  Similar  data  are  shown  in  Table  2  far 
Form  3-H. 


•On  the  basis  of  the  experience  of  this  firm,  it  has  been  found  that 
supervisors  agree  to  a  very  high  degree  in  naming  the  two  or  hree  best 
and  the  two  or  three  least  effective  in  a  group.  This  has  be<-i  found 
to  be  true  in  such  widely  divergent  groups  as  scientists,  doctors  and 
clergymen  in  the  military  service,  scientific  and  technical  man  In 
industry,  supervisors  and  foreman,  hourly-paid  workers  in  industry,  eto. 

4. 
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TABLE  1  “ 


Distribution  of  Snores  Obtained  by  Use  of  Key  1 


Farm 

1-F 

Fora 

2-F  . 

Score 

"Best" 

"Poor" 

"Best" 

"Poor" 

76-50 

72-75 

1 

68-71 

12 

6 

64-67 

22 

32 

60-63 

21 

21 

16 

1 

12 

8 

0 

2 

48-51 

1 

2 

3 

,44-47 

5 

1 

3 

40-43 

5 

6 

136-39 

10 

5 

132-35 

11 

9 

28-31 

8 

6 

24-27 

5 

15  | 

20-23 

13 

13 

16-19 

12 

2 

12-15 

6 

4 

8-11 

1 

3 

4-7 

1 

0-3 

H 

_ 

80 

79 

78 

67 

45.1 

45.7 

18-6 

TABIE  2 

Distribution  of  Scores  Obtained  by  Use  of  Key  1  for  Form  3-1 


Score 

•Best" 

■Poor" 

30- 

28-29 

4 

26-27 

11 

24-25 

21 

22-23 

19 

20-21 

9 

18-19 

9 

16-17 

6 

14-15 

3 

2 

12-13 

3 

10-11 

U 

8-9 

U 

&.7 

1 

4 

4-5 

16 

2-3 

21 

0-1 

11 

H 

_ ?2 _ 

_ 85 _ 

1 
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i  3.  The  distributions  in  Table  1  show  no  sighif leant  difference 

|  between  the  forms  in  either  mean  or  standard  deviation,  and,  in  the 
j  light  of  their  construction  and  validation,  may  be  considered  equivalent 
I  forms  of  the  same  instrument.  No  data  were  available  for  study  of  the 
j  intercarrelations  of  the  three  forms. 


;  4*  One  may  expect  fairly  consistent  scores  from  these  recommend- 

i  ation  forms.  The  decree  of  consistency  or,  more  technically,  the  reli- 
!  ability  of  each  of  the  three  forms  has  been  estimated.  For  Forms  1-F, 

I  and  3-N*  the  estimated  reliability  coefficients  are  ,79>  .84,  and 

j  .77  respectively.  Hie  methods  used  are  described  in  Appendix  E. 
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1.  Richardson,  Bellows,  Henry  and  Company  expresses  its  thanks 
j  to  the  university  physical  science  department  chairmen  for  their  cooper- 
I  ation  in  filling  in  the  Information  Sheets.  The  consistency  of  their 
!  judgments  is  reflected  in  the  cross-VBlidation  of  the  scoring  keys. 

j  2.  Thanks  are  here  expressed  to  those  scientist  supervisors  and 

administrative  officers  who  participated  in  the  validation  of  semi-final 
1  Forms  8,  F,  and  6.  These  persons  are  employed  in  the  following  agencies: 

i  Naval  Research  laboratory 

Ordnance  Research  Laboratory 
|  Naval  Gun  Factory 

;  David  Taylor  Model  Basin 

j  Philadelphia  Naval  Shipyard 

|  Brooklyn  Naval  Shipyard 

Bureau  of  Standards 

!  Brookhaven  National  Institute 

3.  Much  of  the  success  of  the  project  is  due  to  the  efforts  of 
j  Dr.  Ralph  Hogan,  Head,  Manpower  Branch,  Human  Resources  Division,  Office 
of  Naval  Research,  in  his  function  as  liaison  officer. 
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Appendix  A 

PINAL  FORKS  1-P,  2-F,  AND  3-N 
INCLUDING  SUGGESTED  MATERIALS 
FOR  COVES  PAGE  AND  PART  II 


Recommendation  Blank 
Scientific  and  Technical  Positions 


Form  1-f 


v.ditior,  r- 


Title 


Srade 

and  has  given  yo^.r 


Agency  or  Department 

name  as  a  reference.  The  chief  duties  J  the  position  for  which  he  has  applied 


are . 


Please  complete  this  blank  and  return  it  as  soon  as  possible .  Both  Parts  1 
and  II  must  be  filled  out  carefully  and  completely  in  accordance  with  the  di¬ 
rections  if  the  auplicnnt  is  to  receive  due  consideration  for  the  position. 
Use  the  enclosed  self-addressed,  stamped  envelope  to  return  the  completed 
blank  to: 


it  te  "A-  X  V.-  -X 

PART  I*  DESCRIPTION  OP  t>;,;  nan 

Part  I  consists  of  20  blocks  or  groups  of  five  de=cri,ntive  statements.,  Each 
block  has  been  shown  to  discriminate  between  the  more  effective  and  the  less 
effective  physical  scientists  on  the  jet  Jr.  >  < search  laboratories. 

Read  the  statements  in  the  first  block  carefully.  Decide  which  one  of  the 
five  statements  is  MOST  DESCRIPTIVE  cf  the  behavior  of  the  man  as  a  scientific 
worker.  Then,  in  the  column  headed  Most ,  circle  the  letter  corresponding  to 
the  statement  you  select.  Next,  decide  which  of  the  statements  is  LEAST 
DESCRIPTIVE  of  the  mar.  and,  in  the  column  hnadr.o  J.ea3t,  circle  the  letter  that 
goes  with  the  statement. 

Then,  go  on  to  Slock  2  and  the  successive  Hocks  and  do  t  hem  in  the  same  way. 
Answer  every  block.  No  ties  or  omissions  are  allowed. 

*  w  *  *  #  •>.- 

Prepared  bv 

RICHARDSON,  BELLOWS,  HENRY  Sc  CC.,  INC. 

2<U  East  33  Street,  New  York  16,  N.  Y. 


Moat 

Least 

1. 

1  Most  Least 

6. 

A 

A 

Never  satisfied  with  super- 

A 

A 

Devoid  of  ideas. 

ficial  answers. 

B 

B 

Keen  observer. 

B 

B 

Easily  breaks  problems  down 

C 

C 

Gathers  data  before  draw- 

Into  conponent  parts. 

ing  conclusions. 

C 

C 

Wants  to  be  included  in  every 

D 

D 

Quiet  and  retiring. 

activity  and  conversation. 

E 

E 

Has  many  a.jdenta  with 

D 

D 

Uses  more  complex  equipment 

his  apparatus. 

than  necessary. 

K 

E 

Does  not  push  self  or  ideas. 

Most 

Least 

7. 

A 

A 

Avoids  the  practical. 

Most 

Lee  at 

2. 

B 

B 

Doesn't  talk  much. 

A 

Improvises  with  equipment. 

C 

C 

Enjoys  his  work. 

P 

c 

Few  interests,. 

D 

D 

Willing  to  proceed  on 

Helps  suppcrt  himself  with 

tentative  data. 

outside  Jobs. 

E 

E 

Gives  vaguB  answers  to 

i  ' 

D 

lieluctant  to  accept  help  from 

que  stions . 

others. 

l\ 

K 

Needs  little  supervision. 

■ 

Most  Least 

8. 

mm 

A 

A 

Highly  diversified  interests. 

U  - 

l.eaat 

3, 

P 

B 

Adaptable , 

Eft 

A 

Ilea  <  (i  tie  driven  nut  of 

C 

C 

Independent  in  his  thinking. 

laboratory . 

D 

D 

Tends  to  give  too  many 

M 

H 

k««y  t  <i  talk  hi  . 

details. 

1 

■ 

Avoids  the  prar t leal. 

E 

E 

Jokes  about  his  status. 

1' 

(n Iw cm« I avl  in  researches  of 

n Mia  ra . 

l\ 

H 

Iliitmaktnat  tve. 

Most 

Least 

9. 

A 

A 

Modifies  procedure  as  soon  as 

unforeseen  conditions  arise. 

Moat 

1 .*•  x  1 

A. 

B 

B 

Accepts  suggestions  readily 

A 

A 

lur«Jy  looks  for  new  methods. 

and  willingly. 

B 

li 

He  a  nonlng  always  sound, 

C 

C 

Clumsy. 

0 

C 

No  lualma  about  ohallanglng 

D 

D 

Illogical. 

tits  experts. 

E 

E 

Always  looking  for  short  cuts. 

» 

D 

Stubborn  in  argument. 

w 

6 

Never  makes  decisions  until 

he  must. 

Mos  t 

Least 

10. 

A 

A 

He  likes  the  limelignt. 

B 

B 

Practices  good  English  usage. 

Most 

Least 

5. 

C 

C 

Gambles  time  and  energy  on 

A 

A 

Gets  confused  under  pressure. 

long-shot  ideas. 

B 

B 

Raises  provocative  questions 

D 

D 

Always  appears  enthusiastic 

about  theory. 

about  work. 

G 

C 

Very  thorough . 

E 

E 

Accepts  responsibility  only 

D 

D 

Does  not  work  to  limit  of 

when  farced  on  him. 

ability. 

E 

E 

Underestimates  own  ability. 

_ _ _ 

. 

PART  II.  SUMMARY  OF  RECOMMENDATIONS 


Please  summarize  your  recommendations  concerning  this  applicant  by  marking  each 
statement  below  as  follows:  Circle 


♦  if  the  applicant  is  exceptional  or  outstanding  in  the  area. 
0  if  he  is  adequate,  or  about  average  for  a  worker  at  his  level. 
-  if  he  is  deficient,  or  below  average  for  a  worker  at  his  level. 
U  if  you  do  not  know  him  well  enough  to  rate  him  in  the  area. 


1.  Technical  background  and  experience  .  ♦  0 

2.  Executive  ability,  i.e.,  ability  to  formulate  and 

execute  policies,  to  organize  and  direct  the 

work  of  others . ♦  0 

3.  Potential  level  of  development  .  +  0 

4.  Ability  to  get  along  with  people . +  0 

5.  Ingenuity,  creative  ability,  inventiveness  .  +  0 

6.  Educational  background  . . +  0 


U 

u 

u 

u 

u 

u 


REMARKS.  Use  this  space  to  describe  the  applicant  in  any  area  you  have  marked  ♦ 
or  -  above,  giving  specific  examples  of  what  he  has  done  or  failed  to  do  which 
led  you  to  rate  him  as  you  did.  Are  his  major  strengths  and  weaknesses  adequately 
described  above?  Give  any  information  needed  to  make  this  description  of  the  man 
complete  and  pertinent.  Attach  an  additional  sheet  if  more  space  is  needed. 


Please  supply  the  following  information: 

How  long  have  you 

known  the  applicant? _ In  what  capacity? 

./hat  was  your  relationship  to  him? - 

Your  name _ 

College  or  Firm _ 

_<J  re  '.3 - 


Recommendation  Blank 
Scientific  and  Technical  Positions 

Form  2-  f 


Name 


.has  applied  for  a  position  as 


an 


Job  Title 


Grade 

.and  has  given  your 


Agency  or  Department 
name  as  a  reference.  The  chief  duties  of  the  position  for  which  he  has  applied 


are. 


Please  complete  this  blank  and  return  it  a3  soon  as  possible.  Both  Parts  I 
and  II  must  be  filled  out  carefully  and  completely  in  accordance  with  the  di¬ 
rections  if  the  applicant  is  to  receive  due  consideration  for  the  position. 
Use  the  enclosed  self-addressed,  staged  envelope  to  return  the  completed 
blank  to: 


******* 

PART  I.  DESCRIPTION  OF  THE  MAN 

Part  I  consists  of  20  blocks  or  groups  of  five  descriptive  statements.  Each 
block  has  been  shown  to  discriminate  between  the  more  effective  and  the  less 
effective  physical  scientists  on  the  job  in  research  laboratories. 

Read  the  statements  in  the  first  block  carefully.  Decide  which  one  of  the 
five  statements  js  MOST  DESCRIPTIVE  of  the  behavior  of  the  man  as  a  scientific 
worker.  Ti.en,  in  the  column  headed  Most,  circle  the  letter  corresponding  to 
the  statement  you  select.  Next,  decide  which  of  the  statements  is  LEAST 
DESCRIPTIVE  of  the  man  and,  in  the  column  headed  Leant,  circle  the  letter  that 
goes  with  the  statement. 

Then,  go  on  to  Block  2  and  the  successive  blocks  and  do  them  in  the  same  way. 
Answer  every  block.  No  ties  or  omissions  are  allowed. 

******* 

Prepared  by 

RICHARDSON,  BELLOWS,  HENRY  &  CO.,  INC. 

22*  East  33  Street,  New  York  16,  N.  Y. 


Most  Least  1* 

A  A  Aloof. 

B  B  Handles  well  several  Jobs  at 

a  time. 

C  C  Careless  in  handling  equipment . 

0  D  Obtains  needed  information 

from  uncommon  sources. 

E  E  Not  easily  led. 

Most  Least  6. 

A  A  Stubborn  in  argument. 

B  B  Excellent  memory  for  diagrams 

and  drawings. 

C  C  Plenty  of  conn 'ton  sense. 

D  0  Little  intellectual  curiousity. 

E  E  Absent-minded. 

Most  Least  7. 

A  A  Self-centered. 

B  B  Calculations  are  rapid  and 

accurate. 

C  C  Clumsy  in  handling  equipment. 

0  0  Does  not  put  off  doing  things. 

E  E  Always  looking  for  shortcuts. 

Most  Least  2. 

A  A  Mechanical  aptitude  above 

average. 

B  B  Chooses  wisely  between  accuracy 

and  speed. 

C  C  Always  has  the  last  word. 

D  D  Apathetic . 

£  E  Highly  diversified  interests. 

Most  Least  3. 

A  A  Likes  to  work  alone. 

B  B  Thinks  problem  through  before 

seeking  aid. 

C  C  He  is  a  sloppy  housekeeper  in 

the  laboratory. 

DBA  follower  rather  than  a 
leader. 

E  E  Accepts  criticism  in  an  in¬ 

telligent  manner. 

Most  Least  3. 

A  A  Bogged  persistence. 

B  B  Plans  ahead  carefully. 

C  C  Clumsy. 

0  D  Accepted  as  a  leader  among 

associates. 

E  E  Doesn't  see  the  over-all 

picture. 

Most  Least  9> 

A  A  Excellent  mechanic. 

B  B  Tremendous  curiosity. 

C  C  Dislikes  physical  exertion. 

D  D  Fails  to  seek  clarification 

of  instructions  until 
mistake  is  made. 

E  E  Knows  how  to  obtain  coopera¬ 

tion. 

Most  Least  4* 

A  A  Little  insight  into  signifi¬ 

cance  of  problems. 

B  B  Slow  to  find  fault  with  other 

people. 

C  C  Always  carries  out  orders. 

D  0  Lacks  drive. 

E  £  Always  looking  for  shortcuts. 

Most  Least  5  • 

A  A  Takes  as  little  responsibility 

as  possible. 

B  B  Does  well  with  tools. 

C  C  Wastes  time  on  unimportant 

things . 

D  0  Controls  motions  completely. 

E  E  Stresse-  theo.y  over  practice. 

Most  Least  10. 

A  A  Improvises  with  equipment. 

B  B  Over-emphasized  details. 

C  C  Gives  up  easily. 

D  D  V try .abrupt. 

E  E  Studies  a  problem  adequately 

before  making  decisions  or 
conclusions. 

2-F 


Least 

11. 

I  lost  Least 

16. 

A 

Gives  up  even  though  he 

A 

A. 

Evaluates  work  only  in 

thinks  he  is  right. 

terms  of  objectives. 

H 

B 

Be  does  not  evaluate  self 

B 

B 

Willing  to  listen. 

highly  enough. 

C 

C 

Fails  to  foresee  difficulties. 

C 

C 

lakes  the  most  of  souroe 

D 

D 

Barely  looks  for  new  methods. 

materials. 

X 

X 

Willing  to  work  overtime  if 

)D 

D 

Has  poise 

required. 

E 

K 

Frequently  omits  essential 

details  in  a  routine 

I  lost  Least 

17. 

operation. 

A 

A 

Ro  outside  interests. 

O 

D 

Respects  the  knowledge  at 

his  fellow  workers. 

lost  Least 

12. 

C 

c 

Possesses  little  technioal 

A 

A 

Follows  the  group. 

ability. 

B 

B 

Holds  temper. 

D 

D 

Respected  rather  than  liked 

C 

C 

Fails  to  investigate  unex- 

X 

X 

Always  willing  to  consider 

pec  ted  results  in  work. 

suggestions. 

D 

D 

Conventional. 

E 

X 

Always  on  the  job  when  needed. 

lost  Least 

18. 

A 

A 

Insistent  on  neat  work. 

Most  Least 

13. 

B 

B 

Helps  support  himself  with 

A 

A 

Athletic. 

outside  jobs. 

B 

B 

Fakes  nothing  far  granted. 

C 

C 

Inclined  to  be  moody. 

C 

C 

Uses  "lab  jargon"  in  reports. 

D 

D 

Considers  all  aspects  of 

D 

D 

Hesitates  to  offer  new  ideas. 

problems. 

X 

X 

Has  fine  analytical  mind. 

|  X 

X 

Hot  very  bright. 

1  lost  Least 

19. 

lost 

Least 

H. 

A 

A 

Applies  theories  from 

A 

A 

Content  with  average  work. 

other  fields  to  explain 

B 

B 

Always  looking  for  new 

observed  phenomena. 

approaches. 

B 

B 

Follows  line  of  least  re- 

C 

C 

Gives  time  and  attention  to 

sistance. 

small  details  of  procedure. 

C 

C 

Stresses  theory  over  practice , 

D 

0 

Keeps  his  ideas  to  himself. 

D 

D 

Compromises  too  much. 

X 

X 

Too  shy  to  be  a  leader. 

X 

X 

Creative. 

lost  Least 

15. 

lost  Least 

20. 

A 

A 

Incessantly  concerned  with 

A 

A 

Hesitant  in  making  decisions. 

details. 

B 

B 

Performs  necessary  routine 

B 

B 

Independent  in  his  thinking. 

and/or  repetitious  work 

c 

C 

His  conclusions  don't  follow 

without  complaint. 

premises. 

C 

C 

Gets  along  extremely 

0 

D 

A  good  mixer. 

well  with  all  associ- 

X 

X 

Uses  equipment  or  materials 

ates. 

successful  in  related 

D 

D 

Sometimes  off  the  beam  in 

fields  without  specifying 

logic. 

necessary  modifications. 

X 

X 

Sometimes  too  positive. 

2-P 


PART  II.  SUMMARY  OF  RECOMMENDATIONS 


Please  summarize  your  recommendations  concerning  this  applicant  by  marking  each 
statement  below  as  follows:  Circle 


*  if  the  applicant  is  exceptional  or  outstanding  in  the  area. 
0  if  he  is  adequate,  or  about  average  for  a  worker  at  his  level. 
-  if  he  is  deficient,  or  below  average  for  a  worker  at  his  level. 
U  if  you  do  not  know  him  well  enough  to  rate  him  in  the  area. 


1.  Technical  background  and  experience  .  ♦  0 

2.  Executive  ability,  i.e.,  ability  to  formulate  and 

execute  policies,  to  organize  and  direct  the 

work  of  others  ..............  .  *  0 

3.  Potential  level  of  development .  +  0 

4„  Ability  to  get  along  with  people .  +  0 

5.  Ingenuity,  creative  ability,  inventiveness  .  ♦  0 

6.  Educational  background  .....  .  .......  ♦  0 


U 

U 

U 

u 

u 

u 


REMARKS.  Use  this  space  to  describe  the  applicant  in  any  area  you  have  marked  ♦ 
or  -  above,  giving  specific  examples  of  what  he  has  done  or  failed  to  do  which 
led  you  to  rate  him  as  you  did.  Arc  his  major  strengths  and  weaknesses  adequately 
described  above?  Give  any  information  needed  to  make  this  description  of  the  man 
complete  and  pertinent.  Attach  an  additional  sheet  if  more  space  is  needed. 


Please  supply  the  following  information: 

How  long  have  you 

known  the  applicant? - In  what  capacity?. 

What  was  your  relationship  to  him? - 

Your  name _ 

College  or  Firm _ 

Address  - 


Recommendation  Blank 
Scientific  and  Technical  Positions 

Form  3-n 


Name 


has  applied  for  a  position  aa 


in 


Job  Title 


Grade 

and  has  given  your 


Agency  or  Department 

name  as  a  reference.  The  chief  duties  of  the  position  for  itfiich  he  has 
applied  are _ _ _ 


Please  complete  this  blank  and  return  it  as  soon  as  possible.  Both  Parts 
I  and  II  oust  be  filled  out  carefully  and  completely  in  accordance  with 
the  directions  if  the  applicant  is  to  receive  due  consideration  for  the 
position.  Use  the  enclosed  self-addressed,  stamped  envelope  to  return 
the  completed  blank  to: 


******* 

PART  I.  DESCRIPTION  OF  THE  MAN 

Part  I  consists  of  50  statements  which  are  descriptive  of  scientists. 
Rate  the  man  on  the  basis  of  his  performance  as  a  scientist  as  follows: 

In  the  space  at  the  left  of  each  statement,  circle 

■»  if  the  statement  is  a  good  description  of  the  man. 

?  if  the  statement  and  its  opposite  are  about  equally 
true  of  the  man. 

-  if  the  opposite  of  the  statement  is  a  good  descrip¬ 
tion  of  the  man. 

U  if  you  are  unable  to  make  a  judgment. 

Mark  every  item.  Do  not  omit  any  of  them. 

******* 


Prepared  by 

RICHARDSON,  BELLOWS,  HENRY  &  CO.,  INC 
224  East  33  Street,  New  York  16,  N.  Y 


26, 


Willing  to  proceed  on  tenta¬ 
tive  data. 

Knows  how  to  get  a  job  done. 
Conventional. 

Tackles  too  many  problems  at 
one  time. 

Enjoys  being  -vith  others. 


Uses  data  without  understand¬ 
ing  their  limitations. 
Ignores  chance  findings  that 
suggest  new  problems. 
Boorish. 

Clumsy  in  bar -ling  equipment. 
Anxious  to  please  superiors. 


Immense  capacity  for  responsi¬ 
bility. 

Respected  rather  than  liked. 

Work  quality  varies  with 
interest. 

Thrives  on  praise. 

Do'S  nnt  work  to  the  limit  of 
ability. 


Excellent  memory  for  diagrems 
and  drawings. 

Stresses  theory  over  practice. 

Does  not  anticipate  diffi¬ 
culties  th' !  night  arise  at 
later  stages. 

Choo303  wisely  between  accur¬ 
acy  and  speed. 

Considers  consequences  of 
alternative  approach  s. 


Ability  for  analyzing  a 
problem  is  fair. 

Devoid  of  ideas. 

Rather  reserved. 

Hot  adaptable. 

Continues  to  collect  data 
after  conclusiveness  of 
results  is  evident. 


+  ?  -  D 

+  ?  -  U 
+  1  -  U 

*  ?  -  u 

*  ?  -  D 

31. 

Interested  in  researches  of 
others. 

Tremendous  curiosity. 

Willing  to  work  overtime  if 
required. 

Questions  validity  of  ma¬ 
terial  in  the  literatus. 

Tends  to  be  self-centered. 

36. 

+  ?  -  u 

Self-centered. 

+  ?  -  u 

Keeps  opinions  of  other 

people  to  self. 

♦  ?  -  D 

Slow  to  find  fault  with 

other  people. 

♦  ?  -  U 

A  justified  feeling  of 

superiority. 

+  1  -  U 

Brilliant  coordinator  of 

established  ideas,  but  not 

original. 

41. 

♦  ?  -  U 

Very  abrupt. 

+  1  -  U 

Lacks  initiative. 

♦  ?  -  U 

Does  a  minimum  amount  of  work. 

+  ?  -  U 

Doesn't  see  the  over-all 

picture . 

♦  ?  -  U 

Tends  to  let  things  slide. 

Uses  equipment  or  materials 
appropriate  to  related 
fields  without  specifying 
necessary  modifications. 

His  unique  methods  amaze 
supervisors. 

Gets  confused  undor  pressure. 

Somewhat  conceited. 

Organizes  his  work  well. _ 


Turns  out  accurate  work. 

Hesses  up. 

Knows  whether  an  answer  to 
a  mathematical  calculation 
is  reasonable. 

Develops  own  methods  when 
necessary. 

Will  accept  temporary  solu¬ 
tion  to  an  urgent  problem. 

. ^ 


PART  II.  SUMMARY  OF  RECOMMENDATIONS 


Please  summarize  your  recommendations  concerning  this  applicant  by  marking  each 
statement  below  as  follows:  Circle 


*  if  the  applicant  is  exceptional  or  outstanding  in  the  area. 
0  if  he  is  adequate,  or  about  average  for  a  worker  at  his  level. 
-  if  he  is  deficient,  or  below  average  for  a  worker  at  his  level. 
U  if  you  do  not  know  him  well  enough  to  rate  him  in  the  area. 


1.  Technical  background  and  experience  .  +  0 

2.  Executive  ability,  i.e.,  ability  to  formulate  and 

execute  policies,  to  organize  and  direct  the 

work  of  others  . . +  0 

3-  Potential  level  of  development .  +  0 

it.  Ability  to  get  along  with  people .  +  0 

5.  Ingenuity,  creative  ability,  inventiveness  .  *  0 

6.  Educational  background  ......  .  ..  +  0 


U 

U 

u 

u 

u 

u 


REMARKS.  Use  this  space  to  describe  the  applicant  in  any  area  you  have  marked  * 
or  -  above,  giving  specific  examples  of  what  he  has  done  or  failed  to  do  which 
led  you  to  rate  him  as  you  did.  Are  his  major  strengths  and  weaknesses  adequately 
described  above?  Give  any  information  needed  to  make  this  description  of  the  man 
conqolete  and  pertinent.  Attach  an  additional  sheet  if  more  space  is  needed. 


Please  supply  the  following  information: 

How  long  have  you 

known  the  applicant? _ In  what  capacity?. 

What  was  your  relationship  to  him? _ 

Your  name _ 

College  or  Firm _ 

Address  - 


>lch«rdto»,  Btllowt.  Hour?  t  Co.,  Inc. 


Appendix  b 

SAMPLES  OF  MATERIALS  USED  PGR 
ADMINISTRATION  OF 
INFORMATION  SHEETS 
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American  Council  on  Education 

744  JACKSON  PLACE 
WASHINGTON  6.  D  C 


April  5,  1949 


Dear  Professor: 

The  Navy,  in  developing  its  scientific  research 
program,  has  been  interested  in  a  more  effective  method 
of  selecting  its  scientific  and  technical  personnel.  To 
this  end,  the  American  Council  on  Education  has  contract¬ 
ed  with  Richardson,  3ellows,  Henry  &  Co.,  Inc.,  a  nation¬ 
ally  recognized  firm  of  industrial  psychologists,  to  de¬ 
velop  a  more  objective  and  empirically  valid  reconmenda- 
tion  blank  to  be  used  as  an  aid  in  the  selection  of 
scientists . 

A  highly  important  phase  of  this  research  is  the 
present  one,  the  collecting  of  certain  empirical  data  on 
individuals  now  engaged  in  similar  work,  as  a  faculty 
member,  an  advanced  student,  or  the  employee  of  some  busi¬ 
ness  or  government  agency.  We  are  asking  you  to  aid  us 
in  this  work. 

You  are  requested  to  fill  in  the  three  enclosed 
forms,  and  mail  them  in  the  envelope  provided.  Because 
the  work  on  thi"  contract  must  be  completed  by  May  31, 

1949,  your  ironed  ?  attention  to  this  will  be  appreciated. 
These  forms  should  be  filled  out  by  you  or  some  other  mem¬ 
ber  of  your  staff  who  has  had  experience  in  recommending 
science  graduates  for  positions.  The  judgments  should  be 
those  of  an  individual,  not  a  consensus. 

Please  accept  our  thanks  for  helping  in  this  study. 


aJBslb 
Enclosures : 

Information  Sheets  A,  B,  and  C 
Instructions 


Thirty  Mtmd  Annul  Muting,  Mmj  6  •  7,  1949,  H»kl  Majflewtr,  W*.bmgt*n,  D.  C. 


INSTRUCTIONS 


The  purpose  of  the  Information  Sheets  is  to  find  out  how  effec¬ 
tive  the  qualities  reflected  by  the  words  and  phrases  may  be  in  describ¬ 
ing  scientists.  The  information  which  you  contribute  will  enable  us  to 
combine  only  the  more  effective  descriptive  phrases  (as  shown  by  you 
and  others)  into  a  recommendation  form., 

1.  Please  rate  two  of  your  former  students  (either  graduate  or 
advanced  undergraduate)  on  the  appropriate  Information  Sheet.  The  first 
of  these  students  should  be  one  of  the  very  BEST  you  have  had  during  the 
past  five  years.  The  other,  while  completing  the  required  academic  work, 
should  be  one  of  the  POOREST  you  have  had  during  this  sane  period. 

Rate  the  better  of  these  two  students  on  the  Information  Sheet 
stamped  BEST,  and  the  other  on  the  Information  Sheet  stamped  POOR. 

Do  not  name  these  students  on  the  rating  forms,  but  be  sure 
that  you  are  rating  real  persons,  and  not  a  "composite"  student  reflect¬ 
ing  the  attributes  of  many  people. 

The  rating  should  be  done  as  follows: 

In  the  space  at  the  left  of  each  statement 

or  descriptive  phrase,  write: 

5  -  if  the  statement  or  description  fits 
the  person  exactly. 

4  -  if  the  statement  or  description  fits 
the  person  well. 

3  -  if  the  statement  and  its  opposite  are 
about  equally  true  of  that  person, 

2  -  if  the  statement  is  a  rather  poor 
description  of  the  person. 

1  -  if  the  statement  or  description  does 
not  at  all  fit  that  person. 

Make  a  judgment  for  each  item.  If  some 

judgments  are  difficult,  make  a  note  of  it 

at  the  end  of  the  list. 


2.  On  the  third  Information  Sheet,  please  rate  each  of  the  state¬ 
ments  in  terms  of  the  desirability  of  being  so  described.  In  other  words, 
how  "good"  is  it  to  be  called  these  various  things.  Put  your  ratings  in 
the  space  at  the  left  of  each  statement  or  descriptive  phrase  as  follows: 

5  -  if  the  statement  c r  description  could 
be  said  only  of  outstandingly  competent 
or  promising  scientists. 

4  -  if  the  statement  or  description  is  pri¬ 
marily  applicable  to  scientists  or 
students  of  above  average  competence 
or  promise. 

3  -  i.  the  statement  or  description  is  pri¬ 
marily  applicable  to  scientists  or 
students  of  average  competence  or  promise. 

2  -  if  the  statement  or  description  is  pri¬ 
marily  applicable  to  scientists  or 
students  of  less  than  average  competence 
or  promise. 

1  -  if  the  statement  or  description  could  be 
said  only  of  scientists  or  students  of 
no  more  than  minimally  acceptable  com¬ 
petence  or  promise. 

0  -  if  the  statement  or  description  could  be 
said  only  of  individuals  who  so  lack  com¬ 
petence  that  they  cannot  be  called  scientists. 

llakB  a  judgment  for  each  item.  If  some  judgments 

are  difficult,  make  a  note  of  it  at  the  end  of 

the  list. 

3.  As  soon  as  you  have  completed  the  three  Information  Sheets, 
put  them  in  the  enclosed  envelope  and  mail  them  to  Dr.  Harold  A.  gdgerton, 
Richardson,  Bellows,  Henry  and  Company,  Inc.,  224  East  33rd  Street,  New 
York  16,  New  York. 
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Information  Sheet 

Prepared  by 

RICHARDS®,  BELLOWS,  HENRY  &  COMPANY,  ING 
for 

THE  AMERICAN  COUNCIL  ON  EDUCATI® 


Project  RBH-162 — 

Construction  of  an  Experimentally 

Validated  Recomnendation  Blank  to  Project  Director — 

Aid  in  the  Selection  of  Scientists  Harold  A.  Edgerton 

Be  SURE  TO  REA  D  THE  INSTRUCTIONS / 


1 _ Stickler  for  details. 

21  _ Continuous  investigation  made  him 

2 _ Handles  many  types  of  problems  well. 

local  expert  on  special  subject. 

3  ____  Handles  several  jobs  at  a  time  well. 

22  _ Easy  to  talk  to . 

4 _  Does  routine  work  well. 

23  .  Constantly  asks  others  to  solve  his 

5  _  Makes  computational  errors. 

own  problems. 

24  _  Insists  on  working  on  unproductive 

problems . 

6  ___  Evaluates  work  only  in  terms  of  ob- 

25 _ Dogged  persistence.  . J 

jectives. 

7  ___  Pails  to  simulate  practical  conditions 

/ 

in  a  laboratory  test. 

26  _  Needs  to  be  more  decisive./ 

8  ____  Has  trouble  grasping  things  rapidly. 

27  __  Self-centered. 

9 _  Good  at  mental  calculations. 

28  Lets  details/"^ 

10  Employs  graphical  solutions  well. 

29  Fails  to  ^ 

30^" 

11  ___  Usually  gets  the  ri^ht  answer. 

12  __  Efficiency  "expert".  J* 

13  __  Reports  are  the  pocrest  part  of 

work.  y' 

14 _  Apathetic. 

15  __  Slow  to  find  fault  witj>/^ 

16 _  Failure' 

17  — 

SAMPLE  INFORMATION  SHEET 


Richardson,  Bailout*,  Hanry  It  Co.,  Inc. 


Appendix  C 

SAMPLES  OF  MATERIALS  USED  PGR 
ADMINISTRATION  OF 
SEMI-PINAL  PGRMS 
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IINTIMC  MITMODt  IN  PIMONNIL  AND  HUMAN  ABLATIO 

Richardson,  Bellows,  Henry  &  Company 

INeORRORATIB 

1*4  1ART  11*0  IT  HIT.  N  !W  YORK  I*.  N.  Y. 

TILITNONI  HU.  ••I41R 


May,  1949 


Doar  Sir: 

Various  government  agencies  employing  scientific  personnel  are  cooperat¬ 
ing  with  the  Office  of  Naval  Research  and  the  American  Council  on  Educa¬ 
tion  in  a  study  of  the  validity  of  Bthods  of  securing  references  on 
applicants  for  technical  positions.  Scientists  participating  are  asked 
to  rate  tuo  at  their  subordinates  on  the  attached  fonas. 

You  will  note  that  the  task  of  foacpifrflnff  the  person  is  separated  firm 
that  of  evaluating  his  ability.  By  doing  this  it  is  possible  to  objec¬ 
tify  the  evaluation  to  a  degree  not  otherwise  poesible. 

You  are  astasd  to  rate  two  of  your  subordinates.  The  first  of  these,  one 
of  the  best  scientific  workers  you  have  ever  had  under  your  direction, 
ie  to  be  rated  on  the  fern  stamped  "Best."  The  form  stamped  "Poor"  is 
to  be  used  for  the  rating  of  one  at  the  least  adequate  scientific  workers 
you  have  directed.  The  "Poor"  scientist,  ehile  up  to  the  minimum  stand¬ 
ards  of  acceptability,  would  be  one  you  would  just  as  soon  do  without, 
but  still  is  not  poor  enough  to  be  discharged. 

Certain  data  are  required  for  each  individual,  vtio  will  be  otherwise  un¬ 
identified.  Place,  on  the  upper  right  hand  corner  of  the  corer  page, 
ths  person's  professional  level  (e.g,,  P4,  P5,  etc.),  his  field  of  spec¬ 
ialisation,  academic  degrees  held  —  with  major  subject,  the  nuafcer  of 
persons  supervised  directly,  and  the  number  supervised  indirectly. 

Tour  cooperation  is  greatly  appreciated. 


Veiy  truly  yours. 


Vice  President 


HAE’.PH 


Recommendation  Blank 

SCIENTIFIC  AND  TECHNICAL  POSITIONS 


Form  E  (semi-final) 


Prepared  by 

RICHARDSON,  BELLOWS,  HENRY  &  CO.,  INC. 
for 

THE  AMERICAN  COUNCIL  ON  EDUCATION 


Project  RBH-162-- 

Development  of  an  Experimentally 

Validated  Recommendation  Blank  to  Project  Director — 

Aid  in  the  Selection  of  Scientiata  Harold  A.  Edgerton 


This  form  is  a  part  of  the  second  phase  of  an  investigation  of  methods 
for  evaluating  the  abilities  of  applicants  for  scientific  positions. 
The  content  has  been  derived  from  an  analysis  of  the  Inf  on®  ti  on  Sheets 
mailed  to  some  of  you  on  April  5,  1949.  The  farms  will  be  further  re¬ 
fined  and  shortened  as  a  result  of  this  validation  phase  of  the  inves¬ 
tigation. 

There  are  two  sections  in  this  form.  Complete  them  in  order  as  printed. 
Do  not  make  changes  in  the  text  before  responding.  You  will  have  an  op¬ 
portunity  to  make  any  suggestions  for  changes  under  REMARKS  in  the  last 
section,  or  to  include  anything  you  thing  should  be  added. 


SAMPLE  COVER  PAGE  SEMI-FINAL  FORMS 

INSTRUCTIONS  AND  BLOCKS  ARE  SIMILAR  TO  THOSE  IN  FINAL  FORMS  1-F  AND  2~F. 


Scientific  Personnel 
Check  List 

Form  6  (semi-final) 

Prepared  by 

RICHARDSON,  BELLOWS,  HENRY  &  CO.,  INC. 
for 

THE  AMERICAN  COUNCIL  ON  EDUCATION 


Project  RBH-162 — 

Development  of  an  Experimentally 
Validated  Recommendation  Blank  to 
Aid  in  the  Selection  of  Scientists 


This  form  is  a  part  of  the  second  phase  of  an  investigation  of  methods 
for  evaluating  the  abilities  of  applicants  for  scientific  positions. 
The  oontent  has  been  derived  from  an  analysis  of  the  Information  Sheets 
■ailed  to  some  of  you  on  April  5,  1949.  The  forms  will  be  further  re¬ 
fined  and  shortened  as  a  result  of  this  validation  phase  of  the  inves¬ 
tigation. 

This  form  consists  of  fifty  statements  which  are  descriptive  of  scien¬ 
tists.  Bate  the  mac  on  the  basis  of  his  performance  as  a  scientist  as 
follows: 

In  the  space  at  the  left  of  each  statement,  circle 

♦  if  the  statement  is  a  good  description  of  the 
person. 

?  if  the  statement  and  its  opposite  are  about 
equally  true  of  the  person. 

-  if  the  opposito  of  the  statement  is  a  good  de¬ 
scription  of  the  person. 

U  if  you  are  Unable  to  make  a  judgment. 

One  symbol  i3  to  be  circled  for  each  item  If  some 
items  are  difficult,  make  a  note  c£  it  under  REMARKS 
at  the  end  of  the  list. 

SAMPLE  COVER  PAGE  SEMI-FINAL  FORMS 

ITEMS  ARE  SIMILAR  TO  THOSE  IN  FINAL  FORM  3-J* 


Project  Director— 
Harold  A.  Edgerton 


Richardson.  Bellows.  Honry  A  Co..  Inc. 


Appendix  D 

SUGGESTIONS  TO  USING  ACZNCIES 
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ktchcrdion,  Bottom,  Honry  l  Co,  loo. 


!  a.  ffliRQPvmcy 

1.  The  reference  or  recommendation  blank  is  a  device  to  secure 
the  opinion  of  a  qualified  person  in  regard  to  the  suitability  of  an 
applicant  for  a  job.  Reference  blanks  have  ranged  from  a  brief  request 
to  "please  tell  as  about  Bill  Smith”  to  elaborate  series  of  questions  to 
be  answered  regarding  him.  Such  reference  blanks  are  difficult  to  eval¬ 
uate,  frequently  inadequate  in  their  corerage,  full  of  stereotyped  phrases, 
and  frequently  give  little  pertinent  information. 

2.  In  order  to  provide  reference  blanks  which  will  be  less  suscep¬ 
tible  to  the  abcne  criticisms,  three  objective  forms  have  ber;n  constructed: 

Forms  1-F  and  2-F  consist  of  20  blocks  of  forced- 
choice  items. 

Form  3-N  consists  of  50  non- forced  choice  items  of 
i  the  check-list  type. 

t 

i 

|  Part  I  of  these  three  forms  is  in-ended  to  serve  only  as  a  "common  denom¬ 
inator"  in  reference  blanks  to  be  used  for  selection  of  scientific  per- 
j  sonnel.  Other  pertinent  information  must  also  be  obtained  and  evaluated. 
The  suggested  forms  in  Appendix  A  illustrate  some  types  of  materials  that 
might  be  included.  It  is  important  that  Part  I  always  be  used  without 
change .  for  even  minor  modifications  will  seriously  effect  the  validity 
j  and  reliability  of  the  blanks.  Part  II  should  be  modified  to  meet  the 
!  specific  needs  of  the  agency  using  the  fora. 

3.  The  primary  characteristic  of  the  forms  developed  is  that  they 
differentiate  between  the  more  and  less  competent  scientists  and  engineers. 
The  farms  are  adaptable  for  recommendations  by  faculty  members  of  their 
students,  or  by  supervisors  of  scientific  work  in  government  or  industry. 
Finally,  the  forms  are  reasonably  easy  to  use. 

B.  p«  OF  TBS  FORMS 

1.  The  use  and  interpretation  of  any  of  the  three  forme  (1-F, 

2-F,  and  3-N)  depends  greatly  upon  the  purposes  for  which  the  scores  will 
be  used.  In  general,  they  will  be  used  far: 

a.  "Selecting  out,"  that  is,  eliminating  applicants  from 
further  consideration.  When  used  in  this  way,  a  low 
cutting  score  should  be  used  in  order  to  eliminate  the 
most  inadequate  applicants  and  also  to  leave  a  sufficient 
muter  to  allow  for  failures  of  later  hurdles.  This  is 
probably  the  most  inpcrtant  use  of  the  recommendation 
form,  particularly  when  large  numbers  of  applicants  must 
be  considered,  or  then  later  testing  and  interviewing 
involve  considerable  expenditures  of  tine  and  money* 

b.  Selection  of  a  few  applicants  for  intensive  follow-up, 
particularly  idien  only  a  small  number  of  openings  are 
available.  In  this  case,  the  highest-scoring  applicants 
are  considered.  Obviously,  a  careful  screening  is  nec¬ 
essary  in  addition  to  scores  on  the  recommendation  forms. 
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Rlck.rd.on,  follow.,  Henry  ft  Co„  ins. 


It  ia  obvious  that  tha  recommendation  blanka  will  be  most  effective  in 
sslectir*  either  tha  vary  beat  or  tha  vary  pooreat  from  a  group  of  appli- 
canta.  Aa  it  become  a  naoaaaaxy  to  increaae  the  proportion  of  applicant a 
passing  this  hurdle  of  selection,  a  greater  portion  of  persons  of  moder¬ 
ate  ability  will  be  selected.  The  exact  proportions  cannot  be  stated, 
since  the  experimental  s&nplas  ware  restricted  to  the  "best"  and  "poorest" 
subordinates  of  the  reporting  scientists. 

2.  It  is  recommended  that  any  agency  using  any  of  the  forms  1-F, 

2-F,  or  3-N  use  all  of  them.  This  means,  of  course,  that  they  will  have 

three  different  forms  available  for  use.  In  actual  practice,  a  different 
fora  should  be  sent  to  each  person  named  as  reference,  so  as  to  obtain 
a  greater  validity  than  would  be  possible  if  all  the  references  were 
sent  the  same  form. 

C.  INTERPRETATION 

1.  Part  I.  Noras  are  not  available  for  the  scores  on  these  objec¬ 
tive  forms,  although  limited  information  is  given  in  Appendix  E  and  page 

5  of  the  report.  Those  distributions  of  scores  were  obtained  for  a  "best" 

and  a  minimally  satis  factory  group  of  scientists.  Each  using  agency  will 
need  to  accumulate  its  own  experience  and  set  its  own  norms  of  minimum 
scores  in  light  of  i+ s  experience,  policies,  and  purposes. 

2.  Part  II.  Part  II  is  not  objectively  sccrable,  but  the  inform¬ 
ation  given  should  be  carefully  considered.  Persons  waiting  recommenda¬ 
tions  frequently  feel  that  they  have  not  adequately  described  an  applicant 
if  they  have  only  checked  a  carefully  limited  list  of  items,  but  are  will¬ 
ing  to  give  much  meaningful  information  in  response  to  general  questions. 
Also,  it  has  been  shown  that  ratings  of  this  type  are  actually  more  valid 
when  they  follow  the  forced-choice  section. 

3.  Consistency  of  the  reports  can  be  estimted  by  consideration 
of  score  on  Part  I  and  general  tone  on  Part  II.  Thus,  a  recommendation 
would  be  suspect  if  a  person  obtained  an  extremely  high  score  on  Part  I 
and  had  only  minimally  satisfactory  work  history,  training,  background, 
etc.  Similarly,  if  the  objective  score  is  higi,  then  strong  favorable 
statements  regarding  the  applicant  would  be  credible,  but,  if  the  objec¬ 
tive  score  is  low,  a  strongly  favorable  statement  or  tone  in  the  recom- 
mendations  should  be  very  carefully  scrutinized. 

4.  The  distributions  of  scares  in  the  report  (page  5)  and  in 
Appendix  E  reveal  no  significant  difference  between  the  forms  in  either 
mean  score  or  standard  deviation,  and  they  may  therefore  be  considered 
to  be  equivalent  forms  of  the  same  instrument.  No  data  were  available 
for  study  of  the  intercorrelations  of  the  three  forms. 

D.  SCORING  KBTS 

1.  The  strength  of  the  more  objective  type  of  recommendation 
form  lias  in  tha  fact  that  tha  reference  describes  the  applicant  but 
does  not  evaluate  his  descriptions .  Evaluation  is  done  by  naans  of  a 
scaring  key,  and  is  dons  only  by  the  agency  to  whom  the  recommendation 
blank  is  sent.  This  is  essential  since  a  knowledge,  on  the  part  of  the 
reference,  of  the  weights  attached  to  each  response  would  soon  make  the 
blanks  uMlass. 


12 


Richardior.  follow*,  Henry  i  Ca„  Inc. 


2.  Two  keys  ware  constructed  for  each  Torn  —  Kay  1,  a  cross- 
validated  key,  and  Key  2,  an  experimental  key  »hich  needs  further  stutfcr* 

3.  Scoring  keys  are  available  only  to  using  agencies  and  may  be 
obtained  upon  request  to  the  Human  Resources  Division,  Office  of  Naval 
Research. 

s.  Sf  IKS  BLW. 

1.  Any  procedure  or  device  used  in  the  selection  of  personnel 
should  have  tiro  known  characteristics: 

a.  Validity,  i.e.,  it  should  discriminate  between  the  kinds  of 
applicants  desired  and  those  not  so  well  qualified  for  the 
job. 

b.  Reliability,  i.e.,  it  should  produce  consistent  results. 

'  The  validity  and  reliability  of  Part  I  of  these  recommendation  blanks 
j  are  shown  in  Appendix  E.  Both  are  adequately  high.  From  the  experience 
I  of  the  contractor,  it  would  be  expected  that  both  the  validity  and  reli- 
|  ability  of  Forms  1-F  and  2-F  will  remain  slightly  higher  than  for  Form  3-N. 

2.  The  usual  defects  of  ratings  will  appear  to  a  lesser  extent  in 
a  forced-choice  form  than  in  a  non-forced-choice  form. 

F.  SUMMARY  OF  RECOMMENDATIONS 

1.  The  recommendation  blanks  developed  in  this  study  should  be 
adopted  for  use  as  an  aid  in  selection  and  placement  of  scientific  and 
I  engineering  personnel  in  Navy  installations.  While  the  blank  will  be 
;  of  value  in  other  situations,  further  investigation  should  be  made  in 
!  order  to  set  standards  for  minimum  acceptability  of  applicants. 

!  2.  This  device,  as  is  true  of  other  selection  techniques,  can 

be  used  in  at  least  two  ways: 

a.  To  eliminate  only  the  poorest  of  the  applicants,  and 

b.  To  accept  only  the  best  of  the  applicants. 

The  manner  in  which  it  is  to  be  used  will  depend  on  the  policy  of  the 
using  agency.  If  the  score  is  to  be  combined  with  the  sccres  on  the 
remainder  of  the  selection  procedure,  it  will  be  necessary  to  undertake 
the  necessary  investigation  to  determine  the  appropriate  weight  to  be 
assigned  this  device. 

3.  The  c roes- validated  scoring  key  (Key  1)  should  be  used  in 
preference  to  Key  2.  Key  1  is  more  stable  and  has  sufficient  variability 
in  the  scores  fer  any  practical  use.  Key  2  might  be  sihjected  to 
further  study,  but  the  gains  which  will  accrue  from  this  investigation 
are  problematical. 
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a.  BIBLIOGRAPHY 

For  those  who  wish  to  read  further  regarding  these  kinds  of 
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Bills,  M.A.,  "Trends  in  Selection  for  Enployment, "  Personnel. 

1939,  15,  PP.  134-93. 

Knox,  F.M.,  "A  Guide  to  Personnel  Record  Keeping,"  Personnel. 

1 942,  18,  pp.  540-61. 

Newer,  B.S.,  "Enployment  Prospects,"  Personnel  Journal.  1944, 

23,  PP.  135-44. 

Referenses  regarding  fcrced-cholce  performance  reports: 

Bittner,  R.H.,  "Developing  an  Employee  Merit  Rating  Procedure," 
Personnel.  1949,  25,  pp.  275-91. 

Rundquist,  E.A. ,  and  Bittner,  R.H.,  "Using  Ratings  to  Validate 

Personnel  Instruments,"  Personnel  Psychology.  1948,  1,  pp.  163-83. 
Sisson,  E.  Donald,  "Forced-Choice  —  The  New  Army  Rating,"  Personnel 
Psychology.  1948,  1,  PP.  365-81. 

Staff,  Personnel  Research  and  Procedures  Branch,  Adjutant  General's 
Office,  "Forced-Choice  Technique  and  Rating  Scales,"  American 
Psychologist.  1946,  1,  p.  267. 
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Appendix  E 

SUPPLEMENTARY  DATA 

ON  RELIABILITY  AND  VALIDITY  OF  THE  FORMS 


! 
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>  A. 


AL  INFORMATION 


j 

I  The  methods  used  in  the  construction  and  validation  of  the  objec¬ 

tive  fora  of  recommendation  blanks  are  those  which  have  been  developed 
in  connection  with  forced-choice  performance  reports.  Certain  additional 
information  regarding  the  validity  and  reliability  of  the  instruments  is 
presented  in  this  section. 


B.  RELIABILITY 

The  reliability  of  the  three  forms  has  been  estimated,  using  the 
j  split-half  mBthod.  Estimates  are  based  on  both  sets  of  scoring  keys. 
Because  of  the  use  of  contrasting  groups,  split-half  reliability  co¬ 
efficients  were  computed  separately  for  the  "best"  and  "poor"  groups. 

|  The  estimates  shorn  in  the  table  were  obtained  by  taking  the  arithmetic 
mean  of  the  split-half  coefficients  for  the  contrasting  groups  and  then 
I  applying  the  Brown-Spearman  Prophecy  Formula.  These  estimates  of  reli- 
I  ability  are  therefore  probably  quite  conservative. 

i 

!  TABLE  1 


Estimates  of  Reliability 


Form 

Estimated  R 

mmmm 

mm 

1-F 

.79 

.81 

2-F 

.84 

.83 

3-N 

.77 

- 

|  C.  SCORING  KEYS 

Two  scoring  keys  were  developed  far  each  of  Forms  1-F  and  2-F. 

|  Only  one  scoring  key  was  constructed  for  Form  3-N.  Key  1  for  Forms 

i  1-F,  2-F,  and  3-N  was  constructed  in  the  light  of  tho  it  jm  discrimination 
!  shorn  by  the  analysis  of  the  Information  ShBet.  These  scoring  keys  were 
,  applied  to  a  sample  of  government  scientists  far  whom  the  semi-final 
j  fcrms  were  accomplished.  Using  this  different  kind  of  sample  is  a  cross- 
I  validation  of  tto  scoring  key.  The  discrimination  between  "best"  and 
"poor"  groups  were  shown  in  Tables  1  and  2  on  page  5  of  the  report. 

Because  of  the  competition  cf  items  within  ths  relatively  closed 
system  of  forced-choice,  the  analysis  of  the  data  showed  that  additional 
items  in  each  of  the  blocks  could  be  scared.  On  this  basis.  Key  2  was 
developed  for  Forms  1-F  and  2-F.  When  these  keys  were  used  to  score 
the  blanks  in  .to  validation  sample,  there  was  a  significant  increase  in 
i  the  variability  of  tto  scores  relative  to  Key  1  for  Forms  1-F  and  2-F, 
but  there  was  no  significant  change  in  either  of  the  means.  Table  2 
summarizes  the  distribution  of  scores  on  these  keys. 
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TABLE  2 

Distribution  of  Scores  for  the  Validation  Sample 
Using  Key  2  for  Forms  1-F  and  2-F 


Form 

1-F 

1 _ Fora 

2-F _ I 

Score 

"Best" 

"Poor” 

BUSM 

76-80 

17 

16 

72-75 

27 

31 

68-71 

16 

15 

64-67 

11 

5 

60-63 

6 

7 

56-59 

3 

2 

52-55 

1 

1 

48-51 

1 

44-47 

4 

2 

1 

40-43 

2 

l 

36-39 

3 

2 

32-35 

8 

4 

28-31 

3 

6  ! 

24-27 

6 

6 

20-23 

3 

6 

16-19 

13 

9 

12-15 

12 

17 

8-11 

4 

6 

ISP 

16 

2 

4 

5 

N 

80 

79 

78 

67 

U 

44.8 

46.7 

0 

27.6 

27.1 

A  staple  of  52  recommendations  was  excluded  from  all  of  the 
analyses  in  order  to  cross- validate  Key  2  for  Forms  1-F  and  2-F.  While 
this  sample  is  small,  the  results  are  similar  to  those  obtained  with  the 
validation  sample  (Table  2),  and  it  is,  therefore,  possible  to  have  con¬ 
fidence  in  the  validity  of  Key  2  for  Forms  1-F  and  2-F  in  the  scoring  of 
the  reference  blank.  The  scores  for  this  cross-validation  sample  are 
shown  in  Table  3. 
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TABLE  3 

Distribution  of  Scores  for  the  Cross-Validation  Sample 
Using  Key  2  for  Forms  1-F  and  2-F 


Score 

_ _ Eon 

1-F 

1  2-E  .  _ 
"Poor" 

"Best" 

"Poor" 

76-79 

2 

72-75 

2 

68-71 

3 

6^-67 

2 

5 

60-63 

2 

56-59 

1 

1 

52-55 

2 

48-51 

1 

44-17 

1 

40-43 

36-39 

1 

32-35 

1 

28-31 

5 

24-27 

2 

4 

20-23 

3 

2 

16-19 

6 

12-15 

2 

1 

8-11 

1 

1 

4-7 

1 

0-3 

N 

11 

15 

11 

15 

For  each  of  the  semi-final  forms  completed,  the  items  mere 
tabulated  far  professional  level  and  for  professional  field  (physics, 
c hernia try,  mathematics,  engineering).  These  tabulations  showed  almost 
no  differences  among  the  professional  fields.  Distribution  of  items 
checked  shamed  little  difference  for  professional  field. 

Within  the  scope  cf  the  samp  *  studied,  it  is  possible  to  conclude 
that  the  recommendation  blark  can  be  used  to  aid  in  differentiating  the 
more  effective  scientists  from  the  less  effective,  mithout  having  to  use 
a  different  scoring  key  or  different  norms  for  professional  level  or 
professional  field.  It  mas  hoped  that  the  recommendation  farms  mould 
function  in  this  fashion,  letting  detailed  information,  specific  to  a 
field  and  to  professional  level,  be  obtained  hy  questions  designed  to 
obtain  such  information  and  by  study  of  the  applicant's  record. 
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